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Figure 1: Concept sketches are dominated by symmetric strokes both in the shapes they depict, and in the construction lines 
employed to draw these shapes in perspective (a). Our algorithm first decomposes the sketch into locally-symmetric groups 
of strokes (b), and proceeds to identify pairs of strokes that are symmetric with respect to triplets of axis-aligned planes (c). 
At its core, our method selects the stroke correspondences that result in the most symmetric and well connected shape (d). We 
only show a subset of the symmetry planes and correspondences for illustration purpose. 


ABSTRACT 


Concept sketches, ubiquitously used in industrial design, are inher- 
ently imprecise yet highly effective at communicating 3D shape to 
human observers. We present anew symmetry-driven algorithm for 
recovering designer-intended 3D geometry from concept sketches. 
We observe that most concept sketches of human-made shapes are 
structured around locally symmetric building blocks, defined by 
triplets of orthogonal symmetry planes. We identify potential build- 
ing blocks using a combination of 2D symmetries and drawing order. 
We reconstruct each such building block by leveraging a combina- 
tion of perceptual cues and observations about designer drawing 
choices. We cast this reconstruction as an integer programming 
problem where we seek to identify, among the large set of candidate 
symmetry correspondences formed by approximate pen strokes, the 
subset that results in the most symmetric and well-connected shape. 
We demonstrate the robustness of our approach by reconstructing 
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82 sketches, which exhibit significant over-sketching, inaccurate 
perspective, partial symmetry, and other imperfections. In a com- 
parative study, participants judged our results as superior to the 
state-of-the-art by a ratio of 2:1. 
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1 INTRODUCTION 


Concept sketches are commonly used in industrial design, both 
to explore 3D shape ideas during the design process itself and to 
communicate these ideas to colleagues and clients via drawings 
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that leverage human ability to ideate a shape from a set of de- 
scriptive, semantically diverse, yet inexact curves. While humans 
can easily parse a concept sketch, reconstructing a 3D shape from 
such a sketch remains an open problem. A key element in concept 
sketches that artists aim to explore and convey is symmetry. Sym- 
metry is ubiquitous in nature, and even more so in human-made 
environments; whether it is motivated by aesthetics, ergonomics, 
or manufacturability, designers very often create shapes that ex- 
hibit multiple degrees of symmetry". In this paper, we leverage the 
abundance of axis-aligned reflective symmetries in concept sketches 
to automatically lift these sketches to 3D (Figure 1). 

While symmetry has been considered for a number of 3D mod- 
eling tasks [Mitra et al. 2013], detecting and exploiting symmetry 
in real-world concept sketches raises specific challenges as well 
as unique opportunities. On the one hand, concept sketches are 
approximate, incomplete representations of 3D shapes, composed 
of unstructured, swiftly-drawn pen strokes that only form partial 
symmetry correspondences. Notably, many strokes serve as tran- 
sient construction lines that designers draw on and around the object 
to achieve accurate perspective [Gryaditskaya et al. 2019]. These 
construction lines introduce significant visual clutter which further 
challenges the automatic detection of symmetries, as individual 
strokes often have multiple candidate correspondences with respect 
to a given symmetry plane. 

On the other hand, construction lines themselves often repre- 
sent symmetric primitives — also called scaffolds [Gryaditskaya 
et al. 2020; Schmidt et al. 2009] — that approximate the envisioned 
shape and serve as anchors for designers to position smooth sur- 
face curves. Construction lines thus provide a great amount of 
additional symmetry and connectivity cues that we leverage in our 
reconstruction algorithm. They also provide a coordinate system 
that defines the symmetry axes of both the shape and the scaffold. 

Based on the observations above, we propose an algorithm that 
takes as input a concept sketch — represented as a sequence of vector 
strokes recorded with a pen tablet — and jointly identifies symme- 
try correspondences between strokes and deduces the connectivity 
and depth of the 3D drawing’. This formulation combines both 
binary unknowns (which strokes form a symmetry pair, and which 
strokes intersect) and continuous unknowns (where to position the 
strokes and their respective symmetry planes along the depth axis 
to best satisfy the symmetry and intersection relationships). We 
address this combinatorial challenge by casting 3D reconstruction 
as a binary assignment problem. Given a large set of candidate sym- 
metry correspondences, the core of our method selects the subset 
of compatible correspondences that yields the most symmetric and 
well-connected 3D drawing, as measured by a score function that 
we derived from the analysis of designer practices. 

This approach, however, does not generalize as-is to sketches 
of complex shapes which may contain multiple, possibly local, re- 
flective symmetries. Moreover, directly solving for the best 3D 
reconstruction for all input strokes at once is computationally in- 
feasible. We address this challenge by segmenting the inputs into 
locally symmetric building blocks. In constructing the blocks we 
leverage insights about designer workflow. Each constructed block 


1On a random set of 500 models from the ABC dataset [Koch et al. 2019], we found 
that only 4% had a part without any axis-aligned symmetry plane. 
Code and data available at https://ns.inria.fr/d3/SymmetrySketch/ 
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exhibits dense symmetry correspondences with respect to a single 
triplet of axis-aligned symmetry planes. We integrate our binary 
assignment algorithm within a search procedure to find the best 
position of the three planes for each triplet. We then reconstruct 
the blocks in order of appearance in the drawing sequence, such 
that the geometry recovered for a block provides context to anchor 
subsequent blocks. 

We demonstrate the versatility and robustness of our approach 
by reconstructing more than 80 real-world sketches. Our method 
significantly improve over the recent work by Gryaditskaya et al. 
[2020], as validated by a comparative study where participants 
found our results more plausible by a ratio of 2:1. 


2 RELATED WORK 


We focus our discussion on methods that lift a single drawing to 3D, 
especially those that leverage symmetry for this purpose. We refer 
readers to surveys on sketch-based modeling [Bonnici et al. 2019; 
Cordier et al. 2016; Olsen et al. 2009] and symmetry-based geometry 
processing [Mitra et al. 2013] for more general discussions. 

Our method addresses part of the longstanding challenge of as- 
signing depth to pen strokes in a drawing, as pioneered by Lipson 
and Shpitalni [1996]. Since each stroke point can have arbitrary 
depth, additional constraints are necessary to make this problem 
well-posed. Early methods focused on polyhedral shapes where 
many of the strokes are straight, form planar faces, and are parallel 
or perpendicular [Liu et al. 2008; Yang et al. 2013]. Closer to our ap- 
plication domain are methods that leverage the drawing techniques 
of designers, including cross-section lines that depict curvature di- 
rections [Shao et al. 2011; Xu et al. 2014] and construction lines 
that form polyhedral scaffolds around space curves [Schmidt et al. 
2009]. While we benefit from characteristics of design drawings, 
such as the frequent presence of axis-aligned construction lines, we 
do not impose any particular technique requirements on the input. 

We formulate depth estimation as an optimization problem, 
constrained by intersection and symmetry relationships between 
strokes. A complementary trend casts 3D reconstruction as a ma- 
chine learning problem, using large datasets of synthetic drawings 
to train deep neural networks [Delanoy et al. 2018; Guillard et al. 
2021; Li et al. 2018; Wang et al. 2020; Zhang et al. 2021; Zhong 
et al. 2020a,b]. But Gryaditskaya et al. [2019] showed that deep net- 
works trained on synthetic drawings fail on concept sketches due 
to the abundance of construction lines that are not well modeled 
by existing non-photorealistic rendering algorithms. 

Most methods for 3D reconstruction from a single drawing re- 
quire precise input or manual annotations to compute geometric 
relationships between the pen strokes [Chen et al. 2008; Gingold 
et al. 2009; Olsen et al. 2011]. A notable exception is the recent work 
by Gryaditskaya et al. [2020] that lifts real-world concept sketches 
to 3D by identifying whether two strokes that intersect in the draw- 
ing should intersect in 3D, and then deducing the stroke depth from 
these intersections. They solve the resulting binary optimization 
problem using a tailored search algorithm that progresses over the 
drawing sequence to reconstruct small groups of strokes at a time. 
Our approach outperforms this state-of-the-art algorithm by jointly 
identifying intersections and symmetry correspondences. In addi- 
tion to providing strong 3D geometry cues, symmetry allows us to 
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Figure 2: Drawing properties. Concept sketches often ex- 
hibit global and local reflective symmetries with respect to 
axis-aligned planes (a). Designers use existing intersections 
to anchor new strokes, yielding high-valence intersections; 
however, some of the intersections are due to occlusion. Fur- 
thermore, individual lines are often mis-aligned and over- 
sketched (b). 


compute multiple candidate 3D reconstructions for many strokes at 
once, and then to select the best overall reconstruction from these 
candidates using an efficient commercial binary solver. 

Reflective symmetry has long been used as a geometric con- 
straint for sketch-based modeling [Bae et al. 2008; Cordier et al. 
2013, 2011; Miao et al. 2015; Oztireli et al. 2011; Plumed et al. 2016]. 
However, most methods assume the presence of a single global re- 
flective symmetry. Furthermore, existing methods typically attempt 
to identify a small set of confident symmetry correspondences using 
geometric heuristics and then fix these correspondences before mov- 
ing on to 3D reconstruction [Oztireli et al. 2011], or only evaluate 
different possible subsets of correspondences over simple draw- 
ings [Cordier et al. 2013]. These strategies often fail on real-world 
concept sketches that exhibit significant clutter and inaccuracy. In 
contrast, we formulate our problem as an integer program, allow- 
ing us to robustly and efficiently select symmetry correspondences 
based on the quality of the resulting 3D reconstruction. 

Reflective symmetry has also inspired computer vision algo- 
rithms based on multi-view geometry [Koser et al. 2011; Sinha et al. 
2012], or more recently unsupervised learning [Wu et al. 2020]. 
Our formulation is inspired by the work of Jayadevan et al. [2017] 
and Xue et al. [2011], who reconstruct symmetric shapes from pho- 
tographs by selecting symmetry correspondences between edges of 
the image such that the resulting shape is composed of a small num- 
ber of planar faces. Concept sketches, however, contain many more 
lines than photographs because designers frequently draw hidden 
parts of the object, and draw construction lines on and around 
the object. We handle those characteristics of concept sketches by 
formulating a tailored score function that we derive from observa- 
tions about how designers draw, and that is not restricted to planar 
shapes. Our formulation also distinguishes 3D intersections from 
occlusions, which is an additional challenge ubiquitous in design 
sketches due to the presence of hidden and construction lines. 


3 DRAWING PROPERTIES 


Our algorithm is motivated by the key visual cues observers employ 
to recover 3D shape from concept sketches and is designed to 
overcome several challenges posed by such sketches. 
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Symmetry. Most manufactured objects exhibit a global reflective 
symmetry, along with several local symmetries (Fig. 2a). Design- 
ers decompose such objects into axis-aligned geometric primitives 
that are themselves symmetric with respect to triplets of symmetry 
planes (e.g. cuboids and cylinders) [Eissen and Steur 2011; Henry 
2012; Robertson and Bertling 2013]. Designers often construct com- 
plex shapes by drawing one primitive at a time, such that consecu- 
tive strokes depict self-contained locally-symmetric parts. 


Anchoring. To draw lines in perspective, designers often leverage 
existing intersections as anchors through which they trace new lines. 
This strategy results in tightly-connected scaffolds where multiple 
strokes meet at high-valence intersections and most strokes are part 
of two or more intersections [Gryaditskaya et al. 2020] (Fig. 2b). 


Coverage. Designers produce legible sketches by avoiding ex- 
tending strokes beyond their intended length. As a consequence, 
well-anchored strokes typically do not extend beyond their two 
farthest apart 3D intersections [Gryaditskaya et al. 2020]. 


Inaccuracy and Ambiguity. Concept sketches are often over- 
sketched, with multiple intermittent or overlapping strokes depict- 
ing the same line. Concept sketches are also drawn under approxi- 
mate perspective, with pen strokes that do not intersect precisely 
[Gryaditskaya et al. 2020]. Finally, because designers commonly 
draw hidden lines, many intersections are due to occlusion (Fig. 2b). 


4 METHOD OVERVIEW 


Our method takes as input a concept sketch captured with a pen 
tablet, where each pen stroke is represented as a pair consisting of a 
polyline and a time stamp. We assume that the sketch represents a 
shape drawn in perspective, and we apply a vanishing point detec- 
tor to calibrate the perspective camera [Gryaditskaya et al. 2020]. 
Given the camera matrix and a symmetry plane, two strokes that 
are symmetric with respect to that plane can be lifted to 3D via stan- 
dard multi-view geometry, as detailed in supplemental materials. 
Our core reconstruction challenge can therefore be formulated as 
identifying all symmetry planes present in the sketch and detecting 
the pairs of strokes symmetric with respect to these planes. Once all 
such pairs are reconstructed, we use their geometry to reconstruct 
the remaining non-symmetric strokes they intersect. 

While we expect all symmetry planes in our sketches to be 
aligned with the major axes, we do not a priori know the number 
or location of these planes. A key observation behind our method 
is that artists typically draw symmetric strokes soon after each 
other, and mentally break the objects they draw into blocks where 
each block has only one plane of symmetry for each axis; they then 
draw these blocks approximately sequentially, first depicting the 
content of one block and then the next. Based on this observation 
we decompose the drawing into likely symmetric building-blocks, 
where each block exhibits symmetry correspondences with respect 
to a single triplet of axis-aligned planes. We identify these blocks by 
analyzing the local density of symmetry correspondences along the 
drawing sequence. We reconstruct each block in sequence, fixing 
the reconstruction of previous blocks to anchor new ones. 

Even for a single block, the problem of computing the planes 
and the symmetric stroke pairs remains highly challenging. We do 
not know in advance which strokes form symmetry pairs, nor do 
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(c) Concurrent reconstructions due to depth ambiguity 


Figure 3: A stroke can have multiple compatible symmetry correspondences, either because it is symmetric to other strokes 
with respect to several planes (a), or because it is symmetric to several strokes depicting the same over-sketched line (b). Due to 
drawing inaccuracy, the 3D reconstructions given by each correspondence do not perfectly coincide, as depicted in the rotated 
views. A stroke can also have multiple incompatible correspondences, each yielding a very different 3D reconstruction (c). In 
this example, the black stroke could be reconstructed as lying on the front part of the cube due to its correspondence with the 
pink stroke (c, left), or lying far at the back due to its correspondence with the blue stroke (c, right). 


we know where to position the triplet of symmetry planes along 
their respective axes. We tackle this challenge by predicting and 
evaluating multiple plane positions and a large set of candidate 
symmetry correspondences. For each position, we reconstruct all 
candidate pairs of symmetric strokes and select the compatible 
subset that yields the best 3D reconstruction of the entire block. 
We formulate the quality of a reconstruction in the form of a score 
function that favors the presence of multiple symmetries, and an- 
choring of strokes within the block and its predecessors. Combined 
with suitable constraints, this formulation is amenable to efficient 
maximization as an integer program where symmetry correspon- 
dences and stroke intersections are optimization variables. We use 
this formulation to obtain the optimal plane triplet and set of stroke 
pairs symmetric with respect to each of the triplet’s planes. 
Algorithm 1 summarizes the main steps of this approach. In the 
sections below, we first describe the integer program that forms the 
core of our method and which operates on a triplet of axis-aligned 
symmetry planes and a set of candidate stroke correspondences 
computed with respect to these planes (Sec. 5). We then describe 
the details of our overall workflow including how we find the plane 
triplet of each block by searching over the most likely positions of 
axis-aligned planes, and how we segment the drawing into symmet- 
ric building blocks, each associated with a single triplet of planes 
(Sec. 6). The last stage of our method reconstructs non-symmetric 
strokes based on their intersections with symmetric ones (Sec. 7). 
We refer the interested reader to supplemental materials for 
details on how we detect and reconstruct individual candidate 
symmetry correspondences between straight, curved, and elliptic 
strokes. In a nutshell, we consider all stroke pairs that are aligned 
with one of the three vanishing points, and keep the ones for which 
we can match the two strokes closely by reflecting, translating and 
scaling one of the strokes in the direction of the vanishing point. 


5 BLOCK RECONSTRUCTION 


The main component of our method is a block reconstruction proce- 
dure that, given a triplet of symmetry planes and a set of candidate 
symmetry correspondences, identifies the true correspondences to 
deduce the 3D position of their respective strokes. 

Figure 3 illustrates the main challenges we face when attempting 
to solve this task. A single stroke can have multiple symmetry corre- 
spondences with respect to the three planes of the triplet, and each 
correspondence gives a 3D reconstruction of that stroke. But due 


Algorithm 1: Symmetric sketch reconstruction 


C <— Identify candidate symmetry correspondences 
B <— Identify local symmetric building-blocks // Sec.6.1 
sketchReconstruction <— 0 
for Building-block « 8 do 
P <— Identify most likely triplets of planes // Sec.6.4 
bestScore <— 0; bestTriplet — 0; blockReconstruction «— 0 
for Triplet ¢ P do 
// Reconstruct and group compatible correspondences 
S — LiftTo3D(C, Triplet) // Sec.5.1 
// Select the best subset of correspondences 
(Score, Reconstruction) —— 

IntegerProgram(S, sketchReconstruction)// Sec.5 
if Score > bestScore then 

bestScore «— Score; bestTriplet — Triplet 


blockReconstruction <— Reconstruction 
end 


end 
Append(sketchReconstruction, blockReconstruction) 


end 


to drawing inaccuracy, these reconstructions often do not coincide 
perfectly. Furthermore, some candidate correspondences might be 
erroneous and yield reconstructions that are not compatible with 
the reconstructions given by other correspondences. The main goal 
of our algorithm is to identify groups of correspondences that pro- 
duce compatible stroke reconstructions, and to select among these 
groups the ones that yield the best reconstruction of the block. 

We measure the quality of a reconstruction as a function of two 
terms: one measuring the degree of symmetry provided by the 
reconstruction and the other measuring how well connected the 
reconstruction is. The first term depends solely on which correspon- 
dences we select, while the second term also depends on whether 
the 3D strokes reconstructed from these correspondences intersect. 
These terms thus involve two sets of binary variables, one that 
indicates for each candidate correspondence if it is selected, and 
the other that indicates for each stroke intersection if it occurs in 
3D. We formulate this optimization as an integer program, which 
we make efficient by expressing all constraints in a linear form, and 
only one term of the score function in a quadratic form. 
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5.1 Grouping Compatible Reconstructions 


We denote 2D entities with lower-case letters (x), 3D entities with 
upper-case letters (X), and binary variables associated to these enti- 
ties with bold typeface (x, X). Given a candidate pair of symmetric 
2D strokes sp and sg and a corresponding symmetry plane, we 
define their symmetric 3D reconstructions as Spg for sp, and Sgp 
for sg. We denote as Cpq the binary variable that indicates whether 
a pair pq is selected as a symmetry correspondence in the solution. 

Since a stroke sy can have multiple candidate correspondences, 
we first identify which correspondences form compatible groups 
of reconstructions. We denote these groups as Shs k € [1,Kpl. 
We consider two reconstructions Spg and Sp, of a stroke to be 
compatible if they nearly coincide, i.e. if the maximum distance 
between the two reconstructions is below 10% of the length of 
the longest candidate reconstruction of sp. For each group, we 
compute a representative geometry by averaging its constituent 
reconstructions, which we will need later on to detect intersections 
between groups and to evaluate the quality of the correspondences 
selected for each group. 

Equipped with groups of compatible reconstructions for each 
stroke, we then must ensure that the group we select for a given 
stroke is compatible with the groups we select for all other strokes. 
We enforce compatibility of the overall reconstruction by associat- 
ing each group with a binary variable S*, which is equal to 1 if the 
k'h proup is selected. We then add the constraint that at most one 
group should be selected per stroke: >’, sk < 1. We also ensure that 
a group is only selected if at least one of its correspondences has 
been selected, which we express as the constraint si < Yeeck Cc, 


where Gk denotes the set of all correspondences forming group Sk . 
Similarly, a group has to be selected if any of its correspondences 
has been selected, sh >cVceE ce 


5.2 Optimizing for Symmetry 

The first term of our score function favors 3D reconstructions that 
are symmetric with respect to multiple planes in the given triplet. 
To express this property, we associate each stroke sp with a triplet of 
auxiliary binary variables xp, yp, zp, where each variable indicates 
whether there is at least one correspondence cpg that has been 
selected for the corresponding axis-aligned plane. We then seek to 
maximize the sum over all strokes 


Fsymmetry = oy Xp +Yp t+ Zp. (1) 
P 


Due to overdrawing, curves in concept drawings are often drawn 
as a series of contiguous strokes, with each stroke forming partial 
symmetry correspondences with some of the strokes depicting the 
symmetric curve. Simply counting such partial correspondences 
without accounting for the degree of overlap between the strokes 
and the reflections of their counterparts can skew the metric to- 
ward pairs with minuscule partial overlaps. To ensure that the 3D 
reconstruction of a curved stroke is well supported by its partial 
correspondences with other strokes, we measure the overlap be- 
tween each partial reconstruction Spg and S/ 


qP’ 
the reflection of the symmetric reconstruction Sgp with respect to 


, 
where S ap denotes 
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its symmetry plane. We measure the support of curved strokes as 


Fsupport = oy overlap(Spq. Sgp)¢pq- (2) 
Pd 
This term is not necessary for straight strokes because their ge- 
ometry is fully determined by a single correspondence, even if 
partial. 

Finally, since only a subset of the correspondences that form a 
group might be selected, we measure whether this subset is repre- 
sentative of the entire group. We do so by penalizing the maximum 
distance between each selected 3D reconstructions Spq and the 
average geometry of the group: 


Fproximity = » » dist(Sh. Spq)pq: (3) 


k 
pk Cpq eC, 


5.3 Optimizing for Connectivity 


The second part of our score function prioritizes reconstructions 
that follow the principles of anchoring and coverage. To evaluate 
these properties, we first need to identify intersections between 
groups of compatible reconstructions that form the solution. We 
denote a 2D intersection between two strokes sp and sy as ipr, and 
ipr is the binary variable that indicates whether the two strokes 
intersect in 3D. We express ip, in a linear form thanks to symmetric 
range constraints, which only allow an intersection to occur if the 
depth difference between the groups selected for the two strokes is 
small (see supplemental materials for detailed equations). 


Stroke Anchoring. We consider a stroke to be well-anchored if 
it is part of at least two intersections of high valence. To identify 
such intersections, we first group nearby intersections as done by 
Gryaditskaya et al. [2020]. We then consider an intersection to be 
high valence if it is grouped with intersections between strokes 
converging to three different vanishing points, or to two vanishing 
points and to a third arbitrary direction. We count the number of 
high valence intersections along each stroke sp», and activate an 
auxiliary variable wy if the stroke is weakly anchored by being part 
of one intersection, and another variable fp if it is fully anchored by 
being a part of two intersections. We penalize the strokes inversely 
to their degree of anchoring: 


Fanchoring = 3 (2 — Wp — fy) Sp> (4) 
P 
where we activate the binary variable sp if one of the groups of 


compatible reconstructions of sp has been selected, sp = Dix oF 


Coverage. We favor 3D interpretations that cover the input strokes 
well by maximizing the distance between the first and last intersec- 
tions selected along a stroke: 


Feoverage = SS (es tp(i) ~ min wt) Sp, (5) 
Pp P P 


where Jp denotes the set of intersections selected along stroke sp, 
and tp(i) denotes the arc-length parameter value of intersection i. 

To identify the maximum and minimum arc-length parameters 
of selected 3D intersections, we associate each intersection ipy with 
two binary variables, a* , and ab ~» Which respectively specify if the 
intersection should be considered as the first or last intersection 
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Figure 4: We segment the stroke sequence into symmetric 
building blocks based on the density of local symmetry cor- 
respondences (a). We color-code the input strokes by order 
of appearance, from blue to yellow (a,b). In this example, the 
algorithm identifies one block for the bottom cuboid (green), 
one for its rounded edges (orange), and one for the slanted 
cuboid (pink). 


\\ “Ke ‘ 
(a) Input Sketch (b) Global symmetry (c) Auxiliary symmetries 
Figure 5: We first reconstruct all strokes that are symmetric 
with respect to a global plane (b). We then fix this partial 
reconstruction to serve as a scaffold for reconstructing mul- 
tiple local symmetries (c). 


along the stroke. Accompanying constraints ensure that we select 
at most one minimal and one maximal intersection per stroke, that 
each intersection cannot be both minimal and maximal at the same 
time, and that only selected 3D intersections are considered (see 
supplemental materials for detailed equations). Equipped with these 
variables, we rewrite the coverage term as: 


Foovetage = > by tp(ipr apr — » tolipr)apr- (6) 


P ipr€Lp ipr €Lp 


We detail in our supplemental material how we treat the special 
case of ellipses, for which coverage cannot be measured since they 
do not have a start and end; instead, we favor 3D reconstructions 
of ellipses that are most circular. 


5.4 Score Function 


We combine the terms described above to form the score function 
to be maximized subject to the listed constraints: 


Frotal = Fsymmetry + Fsupport + Foroximity + Fanchoring +Freoverage> (7) 


where the binary optimization variables ipg and cpq select intersec- 
tions and symmetry correspondences respectively. We solve this op- 
timization problem with the commercial solver Gurobi [Gurobi Op- 
timization, LLC 2021]. We use a fixed set of weights Asymmetry = 2, 


Asupport = 10, Aproximity = —100, anchoring = 5, and Acoverage =4. 
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6 ALGORITHM DETAILS 


6.1 Computing Building Blocks 


We handle complex sketches by grouping strokes that represent 
largely independent symmetric parts, or building blocks, each with 
its own triplet of symmetry planes. This decomposition breaks the 
reconstruction problem into a series of sub-problems, which con- 
tributes to the scalability of our method and enables us to process 
inputs with multiple symmetry planes per axis. 

We identify self-contained symmetric building blocks by lever- 
aging the observations that their strokes are typically drawn con- 
secutively, and that they share many internal symmetry correspon- 
dences. Treating the input sketch as an ordered sequence of pen 
strokes, we say that a correspondence between stroke sp and sg 
spans over s; if p < i < q. We then count, for each stroke, the num- 
ber of correspondence candidates that span over it. We restrict this 
computation to correspondences between nearby strokes, i.e. those 
that span at most 5 strokes. As illustrated in Figure 4, the resulting 
histogram exhibits distinct modes, one for each highly symmetric 
part. We segment this histogram at each of its local minima to form 
symmetric building blocks. We merge neighboring building blocks 
if they are too small (less than 10 strokes), and we split building 
blocks in half if they are too large (more than 30 strokes). 


6.2 Sequential Block Reconstruction 


We reconstruct the building blocks in their order of appearance in 
the drawing sequence. We use the reconstruction of each building 
block as a scaffold for the next one, such that the selected compatible 
reconstructions act as a geometric context for subsequent strokes 
with which they share symmetric correspondences and intersec- 
tions. We account for this context by computing the score function 
using the strokes of all building blocks processed so far, even though 
we only optimize for the strokes of the current building block. Note 
that while we search for a new triplet of axis-aligned symmetry 
planes for each new building block, we always consider the planes 
used by preceding building blocks so that several building blocks 


can share the same planes if appropriate. 


6.3 Enforcing Globally Symmetric 
Reconstruction 


Typical design sketches contain a mixture of local and global sym- 
metries. We assume and enforce the presence of at least one global 
symmetry plane. We identify this global symmetry by performing 
two passes of our algorithm (Fig. 5). Our first pass enforces global 
symmetry: for each major axis, we place an axis-orthogonal plane 
Telobal at the origin and then run our algorithm with the additional 
constraint that all reconstructed strokes need to have a symmetry 
correspondence with respect to this plane. We keep the best-scoring 
solution across these three axes. This first pass yields a globally 
symmetric, albeit incomplete, reconstruction of the sketch. In a 
second pass, we optimize all remaining strokes while keeping the 
globally-symmetric reconstruction fixed, leaving each stroke free 
to form correspondences with respect to any of the planes. 


Symmetry-driven 3D Reconstruction from Concept Sketches 


6.4 Computing Plane Triplets per Block 


Computing the reconstruction for each block requires a triplet of 
symmetry planes. When only one symmetry plane exists, we can 
fix the scale of the scene by positioning the plane at the origin. 
When a triplet of symmetry planes is present, however, we need to 
solve for the position of each plane along its axis. 

We find the optimal position for a triplet by fixing the first plane 
at the origin, and then searching among candidate positions of 
the two other planes the one that yields the best reconstruction 
overall, as measured by Equation 7. To obtain these candidate posi- 
tions, we first place the three planes at the origin and reconstruct 
all symmetric correspondences of each plane separately. We then 
leverage the fact that translating a plane along its axis is equivalent 
to translating and scaling the 3D reconstruction of the strokes that 
are symmetric with respect to that plane. We loop over each 2D 
intersection ip, and deduce how to position the planes such that 
the reconstructed strokes os and st effectively intersect in 3D. 

Computing a complete reconstruction for each candidate plane 
triplet would be prohibitive. Since the set of 3D strokes recon- 
structed for each plane is sparse, however, only a few triplets yield 
intersections between many strokes. Based on this observation, 
we first compute, for each candidate triplet, the coverage term of 
all candidate reconstructions of all strokes; this gives us an upper 
bound of our score function. We then only run the integer program 
for the positions that maximize this upper bound (we keep the best 
64 positions in our experiments). We further speed up this iterative 
search by providing each run of the integer program with the best 
score obtained so far, allowing the branch-and-bound solver to trim 
any branch of the solution tree that has a lower score. 


7 COMPLETING THE RECONSTRUCTION 


Artist drawings contain strokes with no symmetric counterparts, 
as well as strokes which have such counterparts but which were 
either not matched with these counterparts or assigned different 
sub-par symmetric counterparts by our block-level processing. We 
complete the reconstruction by addressing both types of strokes. 
We improve symmetric correspondences by reevaluating all re- 
constructed strokes with low reconstruction confidence. We define 
a stroke as low confidence if it is part of only one symmetry corre- 
spondence and its coverage term is below 0.5. For any such stroke 
s, we go through all symmetry planes and all other strokes s’; we 
reflect each such stroke around the plane and measure the dis- 
tance between the 2D projection of the reflected stroke and s. We 
reconstruct the stroke s by matching it with s’ if the computed 
projection is within our projection tolerance and the new recon- 
struction improves the coverage of s. We reconstruct the remaining 
non-symmetric strokes using anchoring and coverage cues, lever- 
aging their 2D intersections with previously reconstructed strokes. 
We perform this reconstruction using a greedy version of the al- 
gorithm by Gryaditskaya et al. [2020], which is sufficient in our 
context since few of the non-reconstructed strokes interact. 


8 EVALUATION AND RESULTS 


Fig. 6 compares our results to the ones obtained by Gryaditskaya 
et al. [2020]. Overall, our reconstructions are more faithful to the 
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input sketch, with fewer dangling strokes and non-symmetric dis- 
tortions. We provide additional results and comparisons on 82 
sketches as supplemental materials, in the form of turntable videos. 
These sketches correspond to the subset of first-impression concept 
sketches from OpenSketch [Gryaditskaya et al. 2019] on which we 
managed to calibrate a perspective camera using the algorithm of 
Gryaditskaya et al. [2020]; this calibration failed on 21 sketches due 
to lack of axes-aligned lines. Our algorithm takes a few minutes 
for simple sketches (Fig. 4) to a dozen of minutes for complex ones 


(Fig. 1). 


Comparative study. We conducted a perceptual study to quantify 
our improvement over prior work. We recruited 10 participants, 
which we distributed in two groups that assessed 26 sketches each. 
For each sketch, we presented the input along with our reconstruc- 
tion and the reconstruction by Gryaditskaya et al. [2020]. The two 
reconstructions were colored to highlight their differences, simi- 
lar to Fig. 6. They were displayed in the form of looping videos 
showing slight view changes, and were ordered randomly. The par- 
ticipants were asked to indicate whether the first reconstruction is 
more plausible, the second reconstruction is more plausible, both 
are equally plausible, or both are equally implausible. The plot be- 
low visualizes the 5 answers over all 52 sketches. Our results were 
judged more plausible 51% of the time, while the results of Gryadit- 
skaya et al. [2020] were judged more plausible 23% of the time. Both 
reconstructions were judged equally plausible 14% of the time, and 
equally implausible 12% of the time. We conducted a paired samples 
t-test on these results, which showed that this improvement over 
prior work is highly statistically significant (p-value of 1.0e-07). 


Ours Bothplausible Bothimplausible [Gryaditskaya et al. 2020] 


Limitations. Our method is sensitive to the detection of the ini- 
tial set of candidate symmetry correspondences. If the detection is 
too permissive, the integer program can get overwhelmed by too 
many erroneous correspondences. If the selection is too strict, it 
might miss correct correspondences. We describe in supplemental 
materials the heuristics we use to select likely correspondences. 
Future work might improve on these heuristics, possibly using ma- 
chine learning. Symmetry correspondences are especially difficult 
to detect on long, over-sketched curves drawn under strong fore- 
shortening. We improved the quality of our reconstruction on some 
over-sketched drawings by running the method of Liu et al. [2018] 
to aggregate nearby strokes. But we observed that such filtering 
can degrade results on other sketches as it tends to merge strokes 
that correspond to different parts of the shape. A potential avenue 
for future work would be to also treat stroke aggregation as part 
of the optimization, letting the binary solver decide if two strokes 
should be merged to improve the reconstruction. 


9 CONCLUSION 


We have presented the first algorithm that leverages the abundance 
of multiple axis-aligned reflective symmetries in concept sketches 
to automatically reconstruct such sketches. By casting the joint 
selection of symmetry correspondences and stroke intersections 
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Figure 6: Comparison to the method by Gryaditskaya et al. [2020]. We highlight the strongest differences in red, and provide for 
each sketch the answers from our comparative study (Ours, Gryaditskaya et al. [2020], Both plausible). Our method produces 
fewer dangling strokes (sofa) and can reconstruct disconnected strokes (trash bin, blue strokes). Our results were judged more 
plausible on the majority of cases, although a few were judged inferior, such as the house (right) where our method wrongly 
interprets two diagonal lines as symmetric, which stands out in the otherwise correct reconstruction. 


as an integer program, our method successfully reconstructs real- 
world sketches despite significant clutter and imprecision in such 
input. Our formulation solves for the depth of the pen strokes but 
keeps their position in the drawing plane fixed. Yet, the structural 
information we recover holds great potential to regularize the 3D re- 
construction, for instance by snapping nearby symmetry planes and 
intersections while enforcing the selected symmetry relationships. 
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